Adjoint Algorithmic Differentiation Research Articles

15 years of Adjoint Algorithmic Differentiation (AAD) in finance

Following the seminal ‘Smoking Adjoint’ paper by Giles and Glasserman [Smoking adjoints: Fast monte carlo greeks. Risk, 2006, 19, 88–92], the development of Adjoint Algorithmic Differentiation (AAD) has revolutionized the way risk is computed in the financial industry. In this paper, we provide a tutorial of this technique, illustrate how it is immediately applicable for Monte Carlo and Partial Differential Equations applications, the two main numerical techniques used for option pricing, and review the most significant literature in quantitative finance of the past fifteen years.

A Memory-Efficient Neural Ordinary Differential Equation Framework Based on High-Level Adjoint Differentiation

Neural ordinary differential equations (neural ODEs) have emerged as a novel network architecture that bridges dynamical systems and deep learning. However, the gradient obtained with the continuous adjoint method in the vanilla neural ODE is not reverse-accurate. Other approaches suffer either from an excessive memory requirement due to deep computational graphs or from limited choices for the time integration scheme, hampering their application to large-scale complex dynamical systems. To achieve accurate gradients without compromising memory efficiency and flexibility, we present a new neural ODE framework, PNODE, based on high-level discrete adjoint algorithmic differentiation. By leveraging discrete adjoint time integrators and advanced checkpointing strategies tailored for these integrators, PNODE can provide a balance between memory and computational costs, while computing the gradients consistently and accurately. We provide an open-source implementation based on PyTorch and PETSc, one of the most commonly used portable, scalable scientific computing libraries. We demonstrate the performance through extensive numerical experiments on image classification and continuous normalizing flow problems. We show that PNODE achieves the highest memory efficiency when compared with other reverse-accurate methods. On the image classification problems, PNODE is up to two times faster than the vanilla neural ODE and up to 2.3 times faster than the best existing reverse-accurate method. We also show that PNODE enables the use of the implicit time integration methods that are needed for stiff dynamical systems.

Hessian chain bracketing

Second derivatives of mathematical models for real-world phenomena are fundamental ingredients of a wide range of numerical simulation methods including parameter sensitivity analysis, uncertainty quantification, nonlinear optimization and model calibration. The evaluation of such Hessians often dominates the overall computational effort. The combinatorial {\sc Hessian Accumulation} problem aiming to minimize the number of floating-point operations required for the computation of a Hessian turns out to be NP-complete. We propose a dynamic programming formulation for the solution of {\sc Hessian Accumulation} over a sub-search space. This approach yields improvements by factors of ten and higher over the state of the art based on second-order tangent and adjoint algorithmic differentiation.

15 Years of Adjoint Algorithmic Differentiation in Finance

15 Years of Adjoint Algorithmic Differentiation in Finance

Subdomain separability in global optimization

We introduce a generalization of separability for global optimization, presented in the context of a simple branch and bound method. Our results apply to continuously differentiable objective functions implemented as computer programs. A significant search space reduction can be expected to yield an acceleration of any global optimization method. We show how to utilize interval derivatives calculated by adjoint algorithmic differentiation to examine the monotonicity of the objective with respect to so called structural separators and how to verify the latter automatically.

Efficient computation of nonlinear isogeometric elements using the adjoint method and algorithmic differentiation

We present a consistent and efficient approach to the formulation of geometric nonlinear finite elements for isogeometric analysis (IGA) and isogeometric B-Rep analysis (IBRA) based on the adjoint method. IGA elements are computationally expensive, especially for high polynomial degrees. Using the method presented here enables us to reduce this disadvantage and develop a methodical framework for the efficient implementation of IGA elements. The elements are consistently derived from energy functionals. The load vector and stiffness matrix are obtained from the first and second order derivatives of the energy. Starting from the functional, we apply the concept of algorithmic or automatic differentiation to compute the precise derivatives. Here, we compare the direct (forward) and adjoint (reversed) methods. Analysis of the computational graph allows us to optimize the computation and identify recurring modules. It turns out that using the adjoint method leads to a core-congruential formulation, which enables a clean separation between the mechanical behavior and the geometric description. This is particularly useful in CAD-integrated analysis, where mechanical properties are applied to different geometry types. The adjoint method produces the same results but requires significantly fewer operations and fewer intermediate results. Moreover, the number of intermediate results is no longer dependent on the polynomial degree of the NURBS. This is important for implementation efficiency and computation speed.The procedure can be applied to arbitrary element formulations and coupling conditions based on energy functionals. For demonstration purposes, we present the proposed approach specifically for use with geometrically nonlinear trusses, beams, membranes, shells, and coupling conditions based on the penalty method.

Chebyshev Greeks: Smoothing Gamma without Bias

The computation of Greeks is a fundamental task for risk managing of financial instruments. The standard approach to their numerical evaluation is via finite differences. Most exotic derivatives are priced via Monte Carlo simulation: in these cases, it is hard to find a fast and accurate approximation of Greeks, mainly because of the need of a tradeoff between bias and variance. Recent improvements in Greeks computation, such as Adjoint Algorithmic Differentiation, are unfortunately uneffective on second order Greeks (such as Gamma), which are plagued by the most significant instabilities, so that a viable alternative to standard finite differences is still lacking. We apply Chebyshev interpolation techniques to the computation of spot Greeks, showing how to improve the stability of finite difference Greeks of arbitrary order, in a simple and general way. The increased performance of the proposed technique is analyzed for a number of real payoffs commonly traded by financial institutions.

TurboRVB: A many-body toolkit for ab initio electronic simulations by quantum Monte Carlo.

TurboRVB is a computational package for ab initio Quantum Monte Carlo (QMC) simulations of both molecular and bulk electronic systems. The code implements two types of well established QMC algorithms: Variational Monte Carlo (VMC) and diffusion Monte Carlo in its robust and efficient lattice regularized variant. A key feature of the code is the possibility of using strongly correlated many-body wave functions (WFs), capable of describing several materials with very high accuracy, even when standard mean-field approaches [e.g., density functional theory (DFT)] fail. The electronic WF is obtained by applying a Jastrow factor, which takes into account dynamical correlations, to the most general mean-field ground state, written either as an antisymmetrized geminal power with spin-singlet pairing or as a Pfaffian, including both singlet and triplet correlations. This WF can be viewed as an efficient implementation of the so-called resonating valence bond (RVB) Ansatz, first proposed by Pauling and Anderson in quantum chemistry [L. Pauling, The Nature of the Chemical Bond (Cornell University Press, 1960)] and condensed matter physics [P.W. Anderson, Mat. Res. Bull 8, 153 (1973)], respectively. The RVB Ansatz implemented in TurboRVB has a large variational freedom, including the Jastrow correlated Slater determinant as its simplest, but nontrivial case. Moreover, it has the remarkable advantage of remaining with an affordable computational cost, proportional to the one spent for the evaluation of a single Slater determinant. Therefore, its application to large systems is computationally feasible. The WF is expanded in a localized basis set. Several basis set functions are implemented, such as Gaussian, Slater, and mixed types, with no restriction on the choice of their contraction. The code implements the adjoint algorithmic differentiation that enables a very efficient evaluation of energy derivatives, comprising the ionic forces. Thus, one can perform structural optimizations and molecular dynamics in the canonical NVT ensemble at the VMC level. For the electronic part, a full WF optimization (Jastrow and antisymmetric parts together) is made possible, thanks to state-of-the-art stochastic algorithms for energy minimization. In the optimization procedure, the first guess can be obtained at the mean-field level by a built-in DFT driver. The code has been efficiently parallelized by using a hybrid MPI-OpenMP protocol, which is also an ideal environment for exploiting the computational power of modern Graphics Processing Unit accelerators.

Melting Sensitivities - Exact and Approximate Margin Valuation Adjustments

In this paper we present various methods for fast and accurate calculation of margin valuation adjustments (MVA). We consider the calculation of an MVA using sensitivities for the determination of the initial margin and consider the IDSA SIMM (ISDA Standard Initial Margin Model) as a benchmark. To assess the quality of the various approximations we calculate an - up to Monte-Carlo and discretization errors - exact MVA applying adjoint algorithmic differentiation (AAD). This exact MVA and exact forward initial margin serves as a benchmark for the various approximation methods. It turns out that the conversion from model sensitivities to market rate sensitivities - as required by SIMM - constitutes a computationally expensive calculation step. The step requires the calculation of a (pseudo-) inverse of stochastic matrices. We therefore introduce approximation methods allowing for more efficient MVA calculation.

Adjoint algorithmic differentiation tool support for typical numerical patterns in computational finance

We demonstrate the flexibility and ease of use of C++ algorithmic differentiation (AD) tools based on overloading through application to numerical patterns (kernels) arising in computational finance. While adjoint methods and AD have been known in the finance literature for some time, there are few tools capable of handling and integrating with the C++ codes found in production. Adjoint methods are also known to be very powerful but have potentially infeasible memory requirements. We present several techniques for dealing with this problem and demonstrate them on numerical kernels that occur frequently in finance. We build the discussion around the mature AD tool dco/c++, which is designed to handle arbitrary C++ codes and be highly flexible; however, the sketched concepts can certainly be transferred to other AD solutions including in-house tools. An archive of the source code for the numerical kernels as well as all the AD solutions discussed can be downloaded from an accompanying website. This includes documentation for the code and for dco/c++. Trial licences for dco/c++ are available from Numerical Algorithms Group Ltd.

Source-to-source adjoint Algorithmic Differentiation of an ice sheet model written in C

Algorithmic Differentiation (AD) has become a powerful tool to improve our understanding of the Earth System, because it can generate adjoint code which permits efficient calculation of gradients that are essential to sensitivity studies, inverse problems, parameter estimation and data assimilation. Most source-to-source transformation tools, however, have been designed for FORTRAN and support for C remains limited. Here we use the Adjoinable Land Ice Flow model (ALIF), a C clone of the C++ Ice Sheet System Model (ISSM) and employ source-to-source AD to produce its adjoint code. We present the first running source-to-source adjoint of ALIF, and its application to basal drag inversion under Pine Island Glacier, West Antarctica. ALIF brought several challenges to AD tool development, such as the correct treatment of the context code, which does not compute the differentiable function, but controls this computation through the setup of data structures, including possible aliasing, as well as data-flow reversal in the presence of pointers and dynamic memory, which are ubiquitous in codes such as ISSM and ALIF. We present the strategies we have developed to overcome these challenges.

AAD and least-square Monte Carlo: Fast Bermudan-style options and XVA Greeks

We show how Adjoint Algorithmic Differentiation (AAD) can be used to calculate price sensitivities in regression-based Monte Carlo methods reliably and orders of magnitude faster than with standard finite-difference approaches. We present the AAD version of the celebrated least-square algorithms of Tsitsiklis and Van Roy (2001) and Longstaff and Schwartz (2001). By discussing in detail examples of practical relevance, we demonstrate how accounting for the contributions associated with the regression functions is crucial to obtain accurate estimates of the Greeks, especially in XVA applications.

Automatic Backward Differentiation for American Monte-Carlo Algorithms (Conditional Expectation)

In this note we derive a modified backward automatic differentiation (a.k.a. adjoint automatic differentiation, adjoint algorithmic differentiation) for algorithms containing conditional expectation operators and/or indicator functions. Bermudan option and xVA valuation are prototypical examples. We consider the Bermudan product valuation, but the method is applicable in full generality. Featuring a clean and simple implementation, the method improves accuracy and performance. For conditional expectation operators it offers the ability to use different estimators in the valuation and the differentiation. For the indicator function, the method allows to use per-operator-differentiation of the indicator function, enabling an accurate treatment of each individual exercise boundary - which is not possible in a classic finite difference applied to the Bermudan valuation.

Mini-symposium on automatic differentiation and its applications in the financial industry

Automatic differentiation is involved for long in applied mathematics as an alternative to finite difference to improve the accuracy of numerical computation of derivatives. Each time a numerical minimization is involved, automatic differentiation can be used. In between formal derivation and standard numerical schemes, this approach is based on software solutions applying mechanically the chain rule to obtain an exact value for the desired derivative. It has a cost in memory and cpu consumption. For participants of financial markets (banks, insurances, financial intermediaries, etc), computing derivatives is needed to obtain the sensitivity of its exposure to well-defined potential market moves. It is a way to understand variations of their balance sheets in specific cases. Since the 2008 crisis, regulation demand to compute this kind of exposure to many different case, to be sure market participants are aware and ready to face a wide spectrum of configurations. This paper shows how automatic differentiation provides a partial answer to this recent explosion of computation to perform. One part of the answer is a straightforward application of Adjoint Algorithmic Differentiation (AAD), but it is not enough. Since financial sensitivities involves specific functions and mix differentiation with Monte-Carlo simulations, dedicated tools and associated theoretical results are needed. We give here short introductions to typical cases arising when one use AAD on financial markets.

Risk Optimisation: Finding the Signal in the Noise

We introduce a new method of optimising the accuracy and time taken to calculate risk for a complex trading book, focusing on the use case of XVA. We dynamically choose the number of paths and time discretisation to target computational effort on calculations that give the most information in explaining the PnL of the book. The approach is applicable to both fast, accurate intraday pricing calculations as well as large batch runs. The results are demonstrated by application to a large XVA book, which demonstrates speed-ups comparable to those available via adjoint algorithmic differentiation, for a fraction of the implementation cost.

AAD and Least Squares Monte Carlo: Fast Bermudan-Style Options and XVA Greeks

We show how Adjoint Algorithmic Differentiation (AAD) can be used to calculate price sensitivities in regression-based Monte Carlo methods reliably and orders of magnitude faster than with standard finite-differences approaches. By discussing in detail examples of practical relevance, we demonstrate how accounting for the contributions associated with the regression functions is crucial to obtain accurate estimates of the Greeks for Bermudan-style options and XVA applications.

Adjoint Algorithmic Differentiation and the Derivative of the Cholesky Decomposition

Adjoint Algorithmic Differentiation is an efficient method to calculate the sensitivities of financial instruments to the input parameters. It does require an implementation of the derivative of every elementary step in the pricing algorithm. In particular, for calculating correlation sensitivities in a Monte Carlo simulation it has been proposed to use a step-by-step adjoint derivative of the Cholesky factorization. In this paper we introduce a one-line matrix-level algorithm which uses only matrix inversion and multiplication and can thus be performed by a standard library for linear algebra. Moreover, it allows for a more time-efficient calculation of the standard errors on these sensitivities.

Real-time risk management: An AAD-PDE approach

We apply adjoint algorithmic differentiation (AAD) to the risk management of securities when their price dynamics are given by partial differential equations (PDE). We show how AAD can be applied to forward and backward PDEs in a straightforward manner. In the context of one-factor models for interest rates or default intensities, we show how price sensitivities are computed reliably and orders of magnitude faster than with a standard finite-difference (FD) approach. This significantly increased efficiency is obtained by combining (i) the adjoint forward PDE for calibrating model parameters, (ii) the adjoint backward PDE for derivatives pricing, and (iii) the implicit function theorem to avoid iterating the calibration procedure.

GPU-accelerated adjoint algorithmic differentiation

Many scientific problems such as classifier training or medical image reconstruction can be expressed as minimization of differentiable real-valued cost functions and solved with iterative gradient-based methods. Adjoint algorithmic differentiation (AAD) enables automated computation of gradients of such cost functions implemented as computer programs. To backpropagate adjoint derivatives, excessive memory is potentially required to store the intermediate partial derivatives on a dedicated data structure, referred to as the “tape”. Parallelization is difficult because threads need to synchronize their accesses during taping and backpropagation. This situation is aggravated for many-core architectures, such as Graphics Processing Units (GPUs), because of the large number of light-weight threads and the limited memory size in general as well as per thread. We show how these limitations can be mediated if the cost function is expressed using GPU-accelerated vector and matrix operations which are recognized as intrinsic functions by our AAD software. We compare this approach with naive and vectorized implementations for CPUs. We use four increasingly complex cost functions to evaluate the performance with respect to memory consumption and gradient computation times. Using vectorization, CPU and GPU memory consumption could be substantially reduced compared to the naive reference implementation, in some cases even by an order of complexity. The vectorization allowed usage of optimized parallel libraries during forward and reverse passes which resulted in high speedups for the vectorized CPU version compared to the naive reference implementation. The GPU version achieved an additional speedup of 7.5±4.4, showing that the processing power of GPUs can be utilized for AAD using this concept. Furthermore, we show how this software can be systematically extended for more complex problems such as nonlinear absorption reconstruction for fluorescence-mediated tomography. Program summaryProgram title: AD-GPUCatalogue identifier: AEYX_v1_0Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEYX_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 16715No. of bytes in distributed program, including test data, etc.: 143683Distribution format: tar.gzProgramming language: C++ and CUDA.Computer: Any computer with a compatible C++ compiler and a GPU with CUDA capability 3.0 or higher.Operating system: Windows 7 or Linux.RAM: 16 GbyteClassification: 4.9, 4.12, 6.1, 6.5.External routines: CUDA 6.5, Intel MKL (optional) and routines from BLAS, LAPACK and CUBLASNature of problem: Gradients are required for many optimization problems, e.g. classifier training or nonlinear image reconstruction. Often, the function, of which the gradient is required, can be implemented as a computer program. Then, algorithmic differentiation methods can be used to compute the gradient. Depending on the approach this may result in excessive requirements of computational resources, i.e. memory and arithmetic computations. GPUs provide massive computational resources but require special considerations to distribute the workload onto many light-weight threads.Solution method: Adjoint algorithmic differentiation allows efficient computation of gradients of cost functions given as computer programs. The gradient can be theoretically computed using a similar amount of arithmetic operations as one function evaluation. Optimal usage of parallel processors and limited memory is a major challenge which can be mediated by the use of vectorization.Restrictions: To use the GPU-accelerated adjoint algorithmic differentiation method, the cost function must be implemented using the provided AD-GPU intrinsics for matrix and vector operations. Unusual features:GPU-acceleration.Additional comments: The code uses some features of C++11, e.g. std::shared ptr. Alternatively, the boost library can be used.Running time: The time to run the example program is a few minutes or up to a few hours to reproduce the performance measurements.

Real-Time Risk Management: An AAD-PDE Approach

We apply adjoint algorithmic differentiation (AAD) to the risk management of derivative securities in the situation where the dynamics of securities prices are given in terms of partial differential equations (PDE). With simple examples, we show how AAD can be applied to both forward and backward PDEs in a straightforward manner. In particular, in the context of one-factor short-rate models for interest rates or default intensity processes, we show how one can compute price sensitivities reliably and orders of magnitude faster than with a standard finite-difference approach. This significantly increased efficiency is obtained by combining (i) the adjoint of a forward PDE for calibrating the parameters of the model, (ii) the adjoint of a backward PDE for pricing the derivative security, and (iii) the implicit function theorem to avoid iterating the calibration procedure.